System and method for variation testing for performance test

ABSTRACT

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance testing framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. Each of the variations is composed of a set of logical dimensions and values associated with each of those dimensions to be tested. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

CLAIM OF PRIORITY

This application is a divisional of U.S. patent application Ser. No.11/527,092, filed Sep. 26, 2006, entitled “System and Method forPerformance Testing Framework” by Steven Leslie Roth et al. (AttorneyDocket No. BEAS-01910US1) which claims benefit from U.S. ProvisionalPatent Application No. 60/721,142, filed Sep. 27, 2005 (Attorney DocketNo. BEAS-01910US0).

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following co-pending applicationswhich are hereby incorporated by reference in their entirety:

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FORFLEXIBLE PERFORMANCE TESTING by Steven Leslie Roth and Matthew S.Maccaux filed (Attorney Docket No. BEAS-01911US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FOR FULLRESULTS AND CONFIGURATION STORAGE FOR PERFORMANCE TEST by Steven LeslieRoth and Matthew S. Maccaux filed ______(Attorney Docket No.BEAS-01912US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FORAPPLICATION CONFIGURATION FOR PERFORMANCE TEST by Steven Leslie Rothfiled ______ (Attorney Docket No. BEAS-01913US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FOR ISSUEPATTERN MATCHING FOR PERFORMANCE TEST by Steven Leslie Roth filed ______(Attorney Docket No. BEAS-01914US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FORCENTRALIZED CONFIGURATION AND PROPAGATION FOR PERFORMANCE TEST by StevenLeslie Roth filed ______ (Attorney Docket No. BEAS-01915US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FORPORTAL GENERATOR FOR PERFORMANCE TEST by Steven Leslie Roth filed ______(Attorney Docket No. BEAS-01916US0.

U.S. Patent Application No. ______, entitled SYSTEM AND METHOD FOR ISSUESEVERITY HANDLING FOR PERFORMANCE TEST by Steven Leslie Roth filed______ (Attorney Docket No. BEAS-01917US0.

U.S. patent application Ser. No. 11/527,843, entitled SYSTEM AND METHODFOR PLUGGABLE GOAL NAVIGATOR FOR PERFORMANCE TEST by Steven Leslie Roth,filed Sep. 27, 2006 (Attorney Docket No. BEAS-01919US0).

U.S. patent application Ser. No. 11/528,170, entitled SYSTEM AND METHODFOR GOAL-BASED DISPATCHER FOR PERFORMANCE TEST by Steven Leslie Roth,filed Sep. 27, 2006 (Attorney Docket No. BEAS-01920US0).

U.S. patent application Ser. No. 11/527,962, entitled SYSTEM AND METHODFOR OPTIMIZING EXPLORER FOR PERFORMANCE TEST by Steven Leslie Roth,filed Sep. 27, 2006 (Attorney Docket No. BEAS-01921US0).

U.S. patent application Ser. No. 11/527,967, entitled SYSTEM AND METHODFOR HIGH-LEVEL RUN SUMMARIZATION FOR PERFORMANCE TEST by Steven LeslieRoth, filed Sep. 27, 2006 (Attorney Docket No. BEAS-01922US0).

U.S. patent application Ser. No. 11/527,790, entitled SYSTEM AND METHODFOR QUICK RANGE FINDER FOR PERFORMANCE TEST by Matthew S. Maccaux, filedSep. 27, 2006 (Attorney Docket No. BEAS-01923US0).

U.S. patent application Ser. No. 11/528,064, entitled SYSTEM AND METHODFOR DYNAMIC ANALYSIS WINDOW FOR ACCURATE RESULT ANALYSIS FOR PERFORMANCETEST by Steven Leslie Roth, filed Sep. 27, 2006 (Attorney Docket No.BEAS-01924US0).

U.S. patent application Ser. No. 11/528,010, entitled SYSTEM AND METHODFOR DIMENSIONAL EXPLORER FOR PERFORMANCE TEST by Steven Leslie Roth,filed Sep. 27, 2006 (Attorney Docket No. BEAS-01925US0).

U.S. patent application Ser. No. 11/527,818, entitled SYSTEM AND METHODFOR QUEUED AND ON-DEMAND TESTING FOR PERFORMANCE TEST by Matthew S.Maccaux, filed Sep. 27, 2006 (Attorney Docket No. BEAS-01926US0).

FIELD OF THE INVENTION

This invention relates to the field of performance testing of a deployedapplication software.

BACKGROUND

Performance testing of an application measures one or more of thefollowings:

-   -   Response time: how the application performs when under load,        i.e., what is the average response time during an average load        and what is it during a peak load.    -   Capacity: what the maximum threshold for the application is        under a given set of conditions, i.e., what is the maximum        number of transactions (or pages) per second that the server can        process and how many concurrent users are on the system at this        point.    -   Scalability: how well the application responds to increasing        load (requests to the server) by adding additional resources        (which can be but are not limited to, more CPU's, memories, and        physical boxes), i.e., how does the throughput change as we add        resources and how does the response time change as users are        added to the system.        Most commonly, response time and the throughput of the system        are used as measurements for these terms.

Performance testing of an application can be a daunting and seeminglyconfusing task if not approached with the proper plan in place. Like anysoftware development process, requirements must be gathered, businessneeds should be understood, and a formal schedule should be laid outwell in advance of the actual testing. The requirements for theperformance testing should be driven by the needs of the business andshould be explained with a set of use cases. These can be based onhistorical data (say what the load pattern was on the server for a week)or approximations based on anticipated usage.

Early on in the development cycle of an application, benchmark testsshould be used to determine if there are any performance regressions inthe application. Benchmark tests are great for gathering repeatableresults in a relatively short period of time. The best way to benchmarkis by changing one and only one parameter between tests. For anon-limiting example, the impact of increases in Java Virtual Machine(JVM) memory on the performance of the application can be measured byincrementing the JVM memory in stages (going from say, 1024MB to 1224MB,then to 1524MB and to 2024MB) and stopping at each stage to gather theresults and environment data, record it and then move on to the nexttest. This way there will be a clear trail to follow back when theresults of the tests are analyzed.

Later on in the development cycle of an application, once the bugs havebeen worked out of the application and it has reached a stable point,more complex types of tests can be run to determine how the system willperform under different load patterns. These types of tests are usuallyreferred to as: Capacity Planning, Soak Tests, and Peak-Rest Tests.These tests are designed to test real-world type scenarios by testingthe reliability, robustness, and scalability of the application. For anon-limiting example, capacity planning tests are generally used withslow ramp-ups (defined below), but if the application sees quick burstsof traffic during a period of the day, then the test should certainly bemodified to reflect this. Keep in mind that change of variables in thetest (such as the period of ramp-up or the think-time of the users) willcause the outcome of the test to vary. Therefore, it is always a goodidea to run a series of baseline tests first to set a known controlledenvironment to later compare your changes with.

There are many different ways to go about performance testing of anapplication, some of them more difficult than others. For repeatability,benchmark testing is the best methodology. However, to test the upperlimits of the application in regards to concurrent user-load, capacityplanning tests should be used.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an exemplary performance testing frameworkin one embodiment of the present invention.

FIG. 2 is a flow chart illustrating an exemplary performance testingprocess in one embodiment of the present invention.

FIG. 3 shows a plurality of variations configurable for the performancetesting in accordance with one embodiment of the invention.

FIG. 4 shows an exemplary logical relationship between variations andphases wherein each variation logically runs each phase and can varyanything in any phase.

FIG. 5 shows an exemplary diagram illustrating how the set of phases canbe configured, and how the framework can skip phases, allowingvariations to re-use the output from previous phases.

DETAILED DESCRIPTION

The invention is illustrated by way of example and not by way oflimitation in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

Overall Performance Testing Framework

The present invention enables a performance testing framework thatenables multiple components working together to test a deployedapplication automatically in an unattended manner and to analyze thetest results easily. Here, the application can be but is not limited toa portal of a Web-based service. At very high level, the performancetesting framework can run performance tests with one or more variations(test configurations) on a tested system without user intervention andsave the test results and configuration metadata to a database for lateranalysis. Each of the variations is composed of a set of logicaldimensions and values associated with each of those dimensions to betested . . . Optionally, low-level tools can be run individually as partof a manual testing scenario to cover areas not fully supported by theframework at the high level.

FIG. 1 is an illustration of an exemplary performance testing frameworkin one embodiment of the present invention. Although this diagramdepicts components as functionally separate, such depiction is merelyfor illustrative purposes. It will be apparent to those skilled in theart that the components portrayed in this figure can be arbitrarilycombined or divided into separate software, firmware and/or hardwarecomponents. Furthermore, it will also be apparent to those skilled inthe art that such components, regardless of how they are combined ordivided, can execute on the same computing device or multiple computingdevices, and wherein the multiple computing devices can be connected byone or more networks.

Referring to FIG. 1, a variation controller 100, which optionally can beimplemented based on a JVM, allows a user to interact with and setvarious testing configurations (variations) of two groups of (software)components of the performance testing framework:

-   -   Components (processes to be launched) that are part of the        system 101 being tested under variations, which include but are        not limited to, (managed) Web/application server(s) 102 to        provide Web services to a user, admin server 103 that manages        these Web services, proxy servers (proxies) 104 for the Web        services provided, a load generating tool 105 that generates the        load (test runs) for the performance test and conducts the        performance test of the services, and a performance test        database 106 that stores the test data and configuration of        (repeated) test runs of the performance tests. The mapping of        these tested components to host machines they reside and the        configuration of the database on can be specified via a        configuration file.    -   Components of an infrastructure 110 required to analyze the        performance test results, summarize it across runs, and expose        the data to facilitate analysis and comparisons between runs.        These components include but are not limited to, a results        database 111 that stores the analysis of performance test        results and resides on a different host from the performance        test database; an automation controller 112 implemented via a        state machine that extracts performance test data from the        performance test database 106, analyzes it, and stores the        information in the results database. The automation controller        is also operable to store the results in directories 113 and        present them to the user on a Web browser 114 or export them to        a spreadsheet 115.

FIG. 2 is a flow chart illustrating an exemplary performance testingprocess in one embodiment of the present invention. Although this figuredepicts functional steps in a particular order for purposes ofillustration, the process is not limited to any particular order orarrangement of steps. One skilled in the art will appreciate that thevarious steps portrayed in this figure could be omitted, rearranged,combined and/or adapted in various ways.

Referring to FIG. 2, a plurality of variations of configurationproperties for performance test of a tested system can be specified atstep 201, and the system being tested can be configured automaticallybased on the configuration properties at step 202. Test load of theperformance test can be generated step 203 based on the plurality ofvariations and test runs on the tested system can be conductedautomatically without user intervention at step 204. The performancedata and configuration of the runs can be saved to a test database atstep 205. At step 206, the performance data can be extracted from thetest database and analyzed both within a variation and betweenvariations via a state machine. The analysis results can then be storedin a results database at step 207, and summarized and/or reported to theuser at step 208. Optionally, low-level tools can be run individually aspart of a manual testing scenario to cover areas not fully supported bythe automated test runs at step 209.

In some embodiments, the plurality of variations of the performancetesting can be configured by the variation controller 100 as shown inFIG. 3 with different installers, applications, web-applications,portals, network configurations, domains, startup scripts, databases,proxies, setups (configured portal entities), and/or performance testconfiguration properties (e.g., time, number of concurrent users, etc).For a non-limiting example, the variations of testing runs on anapplication can include changing the installer version and/or setting ofthe application such as portlet type, varying the # or location ofmanaged servers, etc.

In some embodiments, a test run of a variation can either be a ‘ramp-uprun’ or ‘flat run’:

-   -   A ‘ramp-up run’ configures the load generating tool with a        ramp-up schedule that adds additional concurrent user load as        the test runs. For a non-limiting example, the test may start        with 10 users and add 5 users/minute.    -   A ‘flat run’ configures the load generating tool with a schedule        using a fixed number of users and a fixed number of iterations        (test script executions). For a non-limiting example, a typical        test might run with 500 concurrent users, and have each user        perform 6 script iterations. All 500 users start right from the        start.

In some embodiments, a ‘run’ refers to a single performance test,generally including a single launch of load generating tool. A‘singlerun’ variation will cause a single ‘run’ to occur, and a‘multirun’ variation will cause one or more ‘runs’ to occur. The set ofruns associated with a multirun variation is called a ‘flatrunset’,where all runs have a common base configuration and any properties canbe varied between the various runs in a flatrunset, including but arenot limited to, the number of concurrent users and think times.

In some embodiments, a variation can either be a ‘singlerun variation’or ‘multirun variation’.

-   -   With a singlerun variation, the testing framework can test with        either a ramp-up or a flat run.    -   With a multirun variation, the testing framework tests with flat        runs by automatically varying the number of concurrent users in        each test run to identify loading performance, wherein:        -   A ‘divide-and-conquer dispatcher’ multirun can be configured            to explore performance over a range of concurrent users.        -   A ‘goal-based dispatcher’ multirun accepts a set of response            time goals, and tries to find the max number of users the            system can run with while meeting each goal.

In some embodiments, performance data, which can be but is not limitedto, machine loading, proxy stats, session sizes, execute queue lengths,and JVM heap sizes, is collected from various hosts running the systembeing tested while a test run is executing. Server and proxy log filescan be monitored and scanned for issues via pattern match files, whichcategorize and log issues, store log files and detected issues in theresults directory, and can optionally fail the run depending onseverity. As a test runs proceeds, metadata, context, logs, error logs,and results information which can be but is not limited to, allconfigured test run files, server and proxy logs, and performance testreports, are gathered for later detailed analysis.

In some embodiments, the framework is designed to support the followingusage models:

-   -   Run an entire test run automatically.    -   Run part of a test run automatically and part of the test run        manually. For a non-limiting example, one could install and        configure the domain automatically, but configure/build the app        and configure/run the test manually.    -   Run test runs in both ‘everything automatically’ and ‘parts of        it automatically and parts of it manually’ modes. There are two        parts to this:        -   Some configuration properties which the framework can            configure are left unspecified. Many configuration            properties are optional, and if not specified, the framework            will not apply the setting.        -   Not all test configuration properties can be configured by            the framework, so the framework a) does not preclude this,            and b) supports associating these external properties with            the results information. For a non-limiting example, the            framework cannot currently configure every possible setting            in a load generating tool file that the load generating tool            GUI can. Therefore, some configuration properties can be            configured manually via the load generating tool GUI, while            the framework configures various other configuration            properties in the load generating tool file and leaves the            ‘other’ properties as-is. Another non-limiting example is a            Intel Xeon box's ‘hyperthreading’ setting at the BIOS level,            which cannot currently be configured by the framework.            Therefore, a user should be able to tell the framework the            current setting of this in the BIOS, so this setting can get            associated with the results.    -   Manually re-run the load generating tool using test        configurations the framework has created and executed. The        framework uses a ‘template’ and ‘configured copy’ model when        running load generating tool. It copies the ‘original template’        performance test scenario and script files to a results        directory, then configures them in that location, and generates        the performance test analysis directory in that location. Such        an approach allows one to open the performance test scenario        file from the results/scenario directory and re-run the        performance test scenario if desired.    -   Add configuration properties to extend the automation run        capabilities. For a non-limiting example, it would be helpful to        see how certain properties in the configuration file (e.g.        config.xml) affect performance. The framework can be extended to        add this configuration capability, and many others, without much        difficulty.

In some embodiments, the performance testing framework can belaunched/run with several target levels:

-   -   High-level targets, such as ‘local-run-variations’ are used to        run different variations on the performance framework in a        closed-loop, unattended manner by calling low-level targets.    -   Mid-level targets (and some associated custom ant tasks), which        the framework calls to run various phases of testing, which        include but are not limited to, fetch and run a platform        installer; configure and build the application; configure        domain, server script directories, and proxy server; and        configure and run test scenario.    -   Low-level targets (and some associated custom ant tasks) to do        specific tasks, both testing and non-testing related, which        include but are not limited to, fetch and run the platform        installer—‘ant remote-install’; build an application—‘ant        remote-setup-app’; start a single server (or the domain)—‘ant        remote-start-domain’; configure the proxy server; and run        performance tests.

In some embodiments, a Java utility can generate an application tolaunch/run the performance tests. The application can be generated basedon template files and a configuration file (optionally generatingassociated .portlet and .jsp files).

In some embodiments, the performance testing framework can adopt threetypes of communication:

-   -   The variation controller (which runs variations) can invoke a        target on a possibly-remote host, and write ant output to a log        file. It can also take a snapshot of currently-running processes        on a host, which at a later time, can be used to kill any        processes which are new (not in the snapshot). For a        non-limiting example, the main framework Java Virtual machine        (JVM) can invokes target on the proxy box in order to        start/stop/configure the proxy. Even on a single host, the        framework can create structured ant log files, and there is one        ant log file for each phase target during a variation. For a        multirun variation, there is one ant log file for each run        scenarios run within the variation.    -   The variation controller can start an automation        controller/state machine task. For a non-limiting example, the        main framework JVM can create and start a automation controller        task to process the performance test results and create        summary/analysis information in the Oracle results database.    -   The automation controller/state machine can use SSH protocol to        communicate with remote hosts as it runs tasks. For a        non-limiting example, automation controller/state machine can        use SSH to pull results information from the variation        controller host.

In some embodiments, the performance testing framework is operable torecord ant (which is a Java-based build tool) activities and output tolog files for different variations. Such activities include but are notlimited to, configuration metadata, phase errors, and result metadata,and associates them for quick analysis. As the framework completes phasetargets and variations, ant log files are copied into the final resultsdirectory location. This provides both a good way to monitor what theframework is working on, as well as providing a record of what theframework saw during a run.

In some embodiments, the log files are arranged hierarchically, and‘roll up’ output from lower-level log files. There are three primary antlog file locations, which record different levels of output as avariation is processed:

-   -   Console output on the window of the variation controller in        which the run-variations target is launched), which contains        top-level runset (and variation) logging, plus everything below        it.    -   Variation phase target log files that contain ant output for a        particular variation when running a particular phase target. If        any errors were detected during a variation, the framework        generates a ‘ERROR’ file at the top-level results directory.    -   Low-level target output in host-specific directory on the        receiving host side, which contains ant output when a target is        remotely invoked from another a host. In other words, if the        variation controller host invokes and executes a ‘local-install’        target on host B, its ant logging output will be written to host        B's logs directory.

In some embodiments, the performance testing framework provideshigh-level summarizing capabilities to see how changing a test settingaffected performance and errors, and see all runs associated with a testsetting. Such capabilities make it easy to compare test and variationresults, to ‘drill down’ from summarizing information to a flatrun set(group of runs) or single run contained by the summary. In addition,they enable a user to ‘drill down’ to a results directory to see theactual run and error information in order to identify if a run had anerror, and how severe the error is.

Flexible Automated Performance Testing Process

In some embodiments, phase can be used to support changes of any testsetting in a variation, which may include but are not limited to, theinstaller that is used, the application configuration, the proxyconfiguration, database configuration, and/or test configuration. Phasescan be executed either manually or automatically by the performanceframework in the order of:‘install’→‘setupapp’→‘configdomain’→‘runscenarios’The framework can configure, report results and/or issues during any/allof the phases. FIG. 4 shows an exemplary logical relationship betweenvariations and phases wherein each test runs of a variation logicallyruns each phase and can vary anything in any phase.

In some embodiments, one or more of the phases above can be run manuallyor fully automatically and the framework can be configured to onlyconsider certain phases for each variation. While the flexibility makesit possible to change many configuration properties, and determine theireffect on performance, testing variations can be slowed considerably ifeach variation runs every phase (re-install, configure/build app, etc).Thus, new phases can be added and existing phases can be skipped ifdesired in order to achieve faster iterations and save testing time. Fora non-limiting example, the ‘install’ and ‘setupapp’ phases do not needto run every time the system does a test run. A variation can executeone or more phases to apply each configuration, plus one or more testruns with that configuration. The set of phases which must be executeddepends upon what the variations are changing. For another non-limitingexample, for variations with different application configurations butwith the same platform installation, the performance framework wouldneed to configure and build the application (‘setupapp’ phase) as partof testing the performance of each variation. The remaining phases(‘configdomain’, ‘runscenarios’) would also need to be executedautomatically, since they are later in the test sequence. In this case,since the variations are not changing the platform installation, the‘install’ phase does not need to be executed automatically by theperformance framework, and could be run once manually. FIG. 5 shows anexemplary diagram illustrating how the set of phases can be configured,and how the framework can skip phases, allowing variations to re-use theoutput from previous phases.

In some embodiments, each individual configuration of phases can beeither fully specified or be left unspecified, in which case the defaultvalues are used. Such an approach provides an escape hatch forautomation—steps which cannot be automated can be done manually, andused in conjunction with other automated pieces.

-   -   In some embodiments, there are two ways for the framework to        limit the phases that are executed:    -   The framework can be explicitly configured to run only certain        phase targets. This can be done by configuring the property        target.list (an ordered list of phases to run).    -   The framework remembers the properties which were used as each        phase runs. If a new variation would use the same property        configuration when running a phase (such as install) as was used        the last time the phase was executed, the framework will skip        that phase. Optimally, during a framework run, each phase will        execute for the first variation (since the configuration        properties from the last run are unknown), and the framework        will remember the configuration properties for future runs.

In some embodiments, a dispatcher 107 can be launched by the variationcontroller to manages phase execution. Its primary job is to run a groupof phases for each variation, with a set of possibly-overriddenproperties. As it runs the phases, it records information to a ‘resultsdirectory’ on the host file system of the performance test. Since adispatcher knows which phases have executed, it can ‘skip’ a phase ifthe configuration properties are the same as the last time the phase wasexecuted (improves performance of the testing system).

In some embodiments, a dispatcher can use a template to dynamicallygenerate the path to the results directory. It's quite flexible, and canincorporate static text, the value of static or dynamic properties, thevalue of property group tokens being used, date/time stamps, the valueof a label in the variations file, and the variations line itself. For anon-limiting example, running results can be organized in a treestructure by the platform load, or portal size, or think time, orvariation property group tokens, etc. The results directory path isconfigured via a template path with various token types used in thetemplate path.

In some embodiments, a dispatcher can incorporate errormonitoring/detection and handling when running phases, and incorporateserror/issue severity when determining what to do next. While thedispatcher processes a variation or run, if a phase target runs into anerror or an issue, the error/issue will get logged via ant (a Javacomponent tool). Depending on the severity and the framework errorseverity threshold as configured in its properties, an exception mayalso get thrown back to the dispatcher. If this occurs, the dispatcherwill record the exception information in the results directory andadvance to the next variation. This allows the framework to continueprocessing variations even if some variations had errors.

Full Results/Configuration Storage—All Phases

In some embodiments, every component of the performance testingframework-including both the tested system 101 infrastructure 110 isconfigurable. Such configurations include but are not limited to:

-   -   Selecting and installing which version of software, such as a        Web service platform, to test.    -   Generating/configuring the application to be tested, including        specifying which portal/application to use and how to generate        it and its associated XML files such as web.xml and        application.xml.    -   Configuring the servers/proxies providing the test, such as        which application to run on each server, which servers are part        of a test.    -   Configuring the process running on each server, such as domain        configuration/properties, server properties, DB properties, etc.    -   Configuring script/scenario of the plurality of test runs, such        as which test script to run, how many users, etc.

In some embodiments, the configuration can be done either automaticallyor manually and each individual configuration steps can be skipped ifdesired. The manual configuration provides an “escape hatch” forconfigurations that the framework cannot currently automate. For anon-limiting example, a portal can be built manually before it is beingtested by the framework. Alternatively, the portal can be generatedautomatically by the framework before testing, but with somerestrictions.

In some embodiments, configuration metadata can be specified manually,which allows a user to manually configure features that cannot currentlybe automated and make this configuration information available for lateranalysis. For a non-limiting example, a user can manually select adifferent hyper-threading kernel on startup of a Linux server, andinform the framework that the hyper-threading is enabled or not whenlaunching the framework. Such configuration information will beassociated with the test results and can be used during analysis anddata mining just like any other configuration data, even though theframework does not perform this configuration itself.

In some embodiments, all configuration metadata, artifacts and files canbe associated with a performance test (data) and be recorded to supportlater reruns with high repeatability of exact configuration. Suchconfiguration data includes not just the test results, but also theconfiguration of every component in the framework, the context in whichthe results are produced, which can be drilled down later to exactconfiguration data if so desired. In addition, all log files associatedwith the configuration and test can be saved on both the file system andin the database for later analysis if desired. Here, there is nodistinction between the configuration of the framework and test scriptvariation. Overall there can be approximately 500-600 configurationproperties which can be used during automated configuration and thereare several thousand contextual configuration properties which are beingrecorded (artifacts of the system configuration). Such contextualproperties include but are not limited to, portal files used duringtest, all possible test system (e.g., load generating tool) files usedin the test, proxy configuration files used during test, etc. In otherwords, the framework can record as many contextual configurationmetadata as are available, some of which cannot currently be configuredautomatically.

In some embodiments, each component of the performance testing frameworkare pluggable and can be substituted. For a non-limiting example, asystem testing tool can be used in place of the load generating tool orno testing tool is used at all.

In some embodiments, the “performance test” itself is pluggable as theframework can be used for multiple purposes. For a non-limiting example,the framework can skip performance testing and be used for the solepurpose of configuration of the Web servers and proxies. Alternatively,the framework can measure server startup times (or number of errorsreported) as the test result in place of the actual performance of theservers under load test.

Centralized Configuration/Automatic Config Propagation

In some embodiments, the performance testing framework supportscentralized configuration location on the host of the variationcontroller, and the configuration information is then propagated toother hosts of server/proxies in the framework automatically. Such acentralized configuration and propagation approach ensures that theframework properly handles configuration changes (properties can beoverridden in variations) when running variations and it makes itpossible to conduct performance test on servers running on a largenumber of hosts. In addition, this centralized configuration andpropagation approach also enables varying configuration of the serverson the hosts automatically when running tests on them (such runtimeconfiguration would otherwise be impossible otherwise). The primaryfeatures of centralized configuration include, but are not limited to:

-   -   All configuration data is in a single location, regardless of        the tool in which the configuration data is eventually used,        which makes the performance testing framework much easier to use        with large tested systems, such as 50-500 hosts.    -   Centralizing configuration data is based on the ability to        dynamically configure components at runtime, based on the        centralized configuration data. For a non-limiting example,        proxy servers and the tested system can be dynamically        configured at runtime as performance test runs proceed.    -   Centralizing configuration data supports a common mechanism for        dynamically varying configuration data at runtime, such as via        variations and goal-based navigators.    -   Centralizing configuration data supports additional usage        models, such as GUI or web-driven configuration applications.    -   Centralizing configuration data (and storing centralized        configuration data) supports store/restore models for        performance testing. For a non-limiting example, test        configurations can be restored and tests can be re-executed, if        results look suspect.    -   The performance testing framework automatically propagates the        appropriate and current configuration data (possibly including        dynamically modified properties) to each host in the tested        system.    -   Centralizing configuration data supports dynamic varying the        hosts participating in a performance test. For a non-limiting        example, the number of hosts running a server can be dynamically        increased over a series of run being executed by the framework.    -   Centralizing configuration data supports queued/on-demand        testing as discussed later. Since configuration data is stored        separately from the test components, it can be applied to a        different set of test systems on a run-to-run basis.        Issue Pattern Matching and Severity Classification

In some embodiments, the performance testing framework monitors forerrors/issues while it is running performance tests and classifiesindividual issues by severity. More specifically, the framework supportspattern-matching issue detection with a very flexible syntax, and canscan a variety of files that include that are not limited to, installerlog files, database creation output, domain and server log files, proxylog files, and proxy command output for problems by using pattern matchfiles. An error file containing the issue is created when the frameworkdetects an issue when scanning a log file, and context information (logfiles, etc) is saved when an issue is detected. A detected issue mayoptionally affect the behavior of the framework, such as stopping thecurrent variation and advancing to the next one.

In some embodiments, the framework can use pattern match files to scanlog files and output the issues identified. New patterns as well asexclusions can be added by editing pattern matching files. If desired,pattern match scanning can be disabled for certain server and/or proxylog files. In addition, the framework can detect other issue by runningphases in “try catch” blocks, which catch exceptions thrown by (ant)tasks. The framework also performs process monitoring of processes itcontrols; when one of these processes is detected in an incorrect state,an associated issue is created and propagated.

In some embodiments, if the framework finds an issue, it generates anerror file containing the issue and where it was found. Here, an issuecan be identified if a pattern is found (matched) or not found. For anon-limiting example, a Web server domain log file can be scanned forissues with a pattern match file, which searches for the string ‘Error’among other things. If the framework finds one or more ‘Error’ patternmatches in the domain log file, it creates an error file named“domain.errors” in the domain directory with any issues that matched thepattern match file. A more complicated configuration can specificallyexclude certain matches which would normally trigger as issues. For anon-limiting example, if the issue pattern ‘Error’ and the excludepattern ‘FooBar’ are used, the text FooBarError would not be identifiedas an issue, since the exclude pattern matches the text.

In some embodiments, the framework can be configured to scan only asubset of a file (with start and/or end patterns). For a non-limitingexample, it can scan only the part of log file from server running tillserver shutdown starts.

In some embodiments, the framework can classify issues by severity,which starts at 1 (highest severity—phase cannot complete) and decreasesto 10 (lowest severity—minor informational messages). The pattern match(*.pm) file associate a severity with each pattern listed in the file,wherein the severity can be adjusted by editing the pattern match file.If the pattern is detected/matched, an issue of the associated severityis generated. In addition, exceptions thrown by the phases also have anassociated severity.

In some embodiments, pattern match files can use patterns specifiedusing the Java Regular Expressions Pattern regular expression format,where the patterns are specified as partial match patterns. In otherwords, if the pattern is ‘foo bar’ and the line is ‘this is a test foobar some other stuff’, then the pattern matches the line, since thepattern ‘foo bar’ is contained in the line.

In some embodiments, the framework enables detection of ‘when’ the issueoccurred—before startup, during startup, during test, or after shutdowncommenced, based on at least four main pattern types:

-   -   Section patterns: these are patterns (e.g., startsection,        endsection), which if present, specify a sub-section of the file        patterns to be scanned and affect the scope in which the other        patterns are applied. For the following non-limiting example,        -   [[[startsection]]]        -   [[[endsection]]]        -   A request has been received to force shut down of the server        -   Server state changed to SUSPENDING        -   Disconnecting from cluster    -   There is no startsection pattern, so pattern scanning will begin        at the first line. Start section patterns are useful when one is        only interested in the file lines after the line “server        started” occurs, and pattern scanning will continue until the        line “server state changed to SUSPENDING” occurs.    -   Include patterns: these are patterns, which if found, will cause        an issue of the associated severity to be generated (unless        excluded). If a subsection is specified via        startsection/endsection, then only the subsection will be        scanned for include patterns. For a non-limiting example:        -   [[[include severity=“2”]]]        -   (fatal)|(Fatal)|(FATAL)        -   (error)|Error)|(ERROR)        -   (exception)|(Exception)    -   If any of the strings ‘fatal’, ‘Fatal’, ‘FATAL’, ‘error’,        ‘Error’, ‘ERROR’, ‘exception’, or ‘Exception’ are found,        possibly in a section specified by startsection/endsection, and        are not matched by an exclude pattern, then an issue with        severity 2 will be generated.    -   Exclude patterns: these are patterns, which if found, will cause        an issue to not be generated. Generally these are special-case        exceptions to the more general patterns listed in the ‘include’        section. For a non-limiting example:        -   [[[exclude]]]        -   .logErrorsToConsole        -   Failed to roll HTTP log file for the Web server        -   Failed to rename log file on attempt to rotate logs    -   Suppose a line contains the text ‘.logErrorsToConsole’, and        suppose an include pattern ‘Error’ is specified. Since the line        contains the include pattern ‘Error’, the include pattern would        normally generate an issue. But by specifying the exclude        patterns listed above, an issue will NOT be generated.    -   Required patterns: these are patterns which are expected to be        present. If not found, an issue of the associated severity is        generated. For a non-limiting example:        -   [[[required severity=“2”]]]        -   Server started in RUNNING mode    -   If no matches are found for the specified pattern, then an issue        of severity 2 is generated.        Issue Severity Handling

In some embodiments, the framework can incorporate issue detection andhandling (or ignoring) issues throughout the automated performancetesting process (performance test issues, such as failed transactions,are handled separately). Any issues detected are handled based on theirseverity, which enables the issues to control the performance test runprocess. Such control may include but is not limited to, record resultsbut continue running the test run, stop current run and advance to nextrun (for multirun variations), and stop current run, advance to nextvariation (for singlerun variations). Issues higher than a certainseverity are automatically recorded and associated with the run results.If one tries to configure the framework to run tests that may not bevalid, the framework can detect any issues which occur, properly resetthe system and move on to the next variation.

In some embodiments, the framework is operable to handle issuesaccording to the following two configured severity thresholds, and anydetected issues (beyond the specified severity threshold) are associatedwith the results and are visible during analysis:

-   -   Framework log severity threshold: which specifies the severity        threshold at or below which issues should be recorded in the        results database. This is used to filter error data storage to        only the highest priority errors. For a non-limiting example, if        threshold is set to 10, then all issues of severity 10 or below        (1 . . . 10) will be recorded in the results database. Any        issues are logged to the log file regardless of how this is        configured.    -   Framework issue severity threshold: which specifies the severity        threshold at or below which errors should throw an exception.        This controls the issue priority which will the framework        advance to the next run. An exception will generally cause the        framework to stop the current run abruptly and advance to the        next one, which generally means the current run will not get        recorded in the results database. If an exception is not thrown,        then the framework will continue processing the current run. For        a non-limiting example, if the threshold is set to 3, then all        errors of severity 1, 2, or 3 will cause an exception to be        thrown, which will cause the current run to not get recorded in        the results database.

In some embodiments, framework log severity threshold should beconfigured to a high number (so any errors during the runscenarios phaseare recorded to the database), while framework error severity thresholdshould be configured to a low number (0 or 1) if error runs are berecorded in the results database, or if the framework is to ignore anyissues it detects and keep running.

System Process Configuration and Processing Infrastructure (Net ConfigFile)

In some embodiments, the performance testing framework can accommodatemany different host-specific network configurations of the system beingtested. Such network configuration capability provides the foundationfor capacity (load) testing of components located on a large number ofhosts. More specifically, it allows configuration of which components torun on each host by specifying a mapping of running processes(components) to hosts. Such mapping includes but is not limited to,which hosts are part of the system being tested, which host(s) theserver instances (e.g., admin server and Web servers) will run on,directory of component locations on each host, and how each component isconfigured on it. It can also include the configuration of the loadgenerating tool, and any proxy servers

In some embodiments, the framework automatically configures and managesprocesses based on the configuration properties in network configurationfile, which describes the network configuration and any host-specificand server-specific configuration properties of the tested system 101.The network configuration file can be a simple 3-level XML file:

-   -   The top level element, <networkConfig>, groups the hosts of the        tested system.    -   The mid level element, <host>, groups tested components        (servers) placed (run) on a host.    -   The bottom level elements represent each component running or        placed on that host. More specifically, the network        configuration file describes the location (and potentially some        host-specific/server-specific configuration) of at least the        following tested system components: admin server, Web server(s),        proxy server(s), load generating tool, etc. For each server, the        file may specify its individualized configuration properties,        which include but are not limited to, memory and JVM to use, and        which ports of the host to listen to. For each proxy server, the        file may specify which type of proxy server to use, which host        they will be run, etc.

In addition to the test configuration in the network configuration file,there are several hosts/servers for the performance test infrastructurenot listed in the network configuration file, which the framework needsto know about. These configured as load test properties include but arenot limited to: automation controller, performance test DB, and OracleResults Database

In some embodiments, a typical network configuration of the performancetesting framework includes at least the following:

-   -   One primary host, this is where the variation controller runs,        the framework control and configuration occurs, and the loading        generating tool and analysis runs.    -   One or more load generating tools, if needed.    -   A deployed admin server.    -   Deployed Web server(s), and    -   Optionally deployed proxy server(s).

In some embodiments, the framework can run performance tests withvarying network configuration of the tested system across the runs tosee how various types of hosts can affect performance. For anon-limiting example, the tested system can be configured to run anumber of tests (variations), using one server per host with variousnumbers of hosts (e.g., 1, 2, 3, 5, 10, 20, 30) hosts to see how thesystem scales as hosts are added. Alternatively, the system can beconfigured to run variations varying both the number of hosts and thenumber of servers per host, e.g.,

-   2 hosts, 1 server/host==>2 total servers-   2 hosts, 2 server/host==>4 total servers-   2 hosts, 3 server/host==>6 total servers-   . . .-   4 hosts, 1 server/host==>4 total servers-   4 hosts, 2 server/host==>8 total servers-   4 hosts, 3 server/host==>12 total servers    Such an approach can be used to determine how the tested system    scales as additional servers are added to individual hosts. Beyond a    certain point, one would likely find it does not make sense to add    any more servers to a host as additional host(s) would be needed.    Portal Generator

In some embodiments, the performance testing framework utilizes aportal/portlet generator to simplify portal configuration testing. Morespecifically, the portal/portlet generator allows portal configurationto be automatically created and used as part of the performance test.For a non-limiting example, it allows performance tests to be run with1, 2, 5, 10, 50, 100 portlets/page.

In some embodiments, the portal generator allows portals to be generatedbased on a ‘shape’, i.e., a specified number of books/pages/portletsspecified in one or more logically-identical portal file(s). Here,‘logically-identical’ means the files have the same number ofbooks/pages/portlets, though the instance IDs (and possibly definitionIDs) are different. For a non-limiting example, the portal/portletgenerator can be used to generate a portal file which is 3 books deep,each leaf book containing 2 pages, and each leaf page containing 10portlets and report the total number of books/pages/portlets created.

In some embodiments, the portal generator incorporates various templatefiles and tree configuration properties for flexible portal generation.There can be one template file for the start and the end of each ‘node’in the portal tree. For non-limiting examples, there are book header andportlet footer files used at the appropriate time, which may include oneor more of, book start, book end, portlet start, portlet content,portlet end, page start, page end, portal start, and portal end. Foranother non-limiting example, there is a portlet template file, whichdefines the .portlet file and any other files it references. If it is aJSP portlet, then there will also be one or more JSP files (for content,for edit, for help, etc).

In some embodiments, a portlet template directory specifies what theportlets look like if portlet generation is enabled. The portlettemplate directory holds template files which are laid down for eachportlet that the portal generator creates, each generated portlet isplaced in its own directory, with associated files. The directoryincludes portal files which are added at various times when building upthe portal file, wherein each file specifies an XML fragment which isadded at the appropriate time by the portal generator when constructinga portal file. For a non-limiting example, as the portal generatorstarts the portal, it adds the contents of the file named‘portal.template.portalheader’. When it starts a book, it adds thecontents of the file named ‘portal.template.bookheader’. When itcompletes a portlet, it adds the contents of the file named‘portal.template.portletfooter’. The template files can includedynamically-substituted text as well—for non-limiting examples, the‘portal.template.bookheader’ file contains the text ‘${book.default.page}’, ‘$ {book.definition.label }’, ‘$ {book.title}’,and ‘${book.menu}’ which are replaced by the portal generator withdynamic text when it adds the book header file.

In some embodiments, the portal generator can either perform portletgeneration for each individual portlet instance (1 per portlet), whereeach portlet instance would reference its own generated portlet file(portlet definition) and each .portlet file would reference its owngenerated JSP file for content; or use a shared instance, where eachportlet instance can be configured to share a manually-created singleportlet file and associated JSP.

In some embodiments, token substitutions can be performed by the portalgenerator and can be used within the template files. These tokensubstitutions include but are not limited to, ${book.default.page},${book.definition.label}, ${book.menu}, ${book.title},${page.definition.label}, ${page.title}, ${portal.name}, ${portlet.contenturi}, ${portlet.instancelabel}, ${portlet.title}, and${tree.optimization.attribute}.

In some embodiments, a portal tree reflecting the portal configurationcan be logically constructed from nonleaf nodes (books/pages andportlets within a page) and leaf nodes (pages and portlets within thepage), wherein nonleaf node template can be configured and multiplereplicas will be created. The same goes for the leaf node template. Thena tree is generated based on the templates, a tree summary is dumped,and the utility writes out the portal file and optionally, the .portletfiles and associated content.

In some embodiments, a sample configuration of a portal can include atleast the following attributes: book depth (number of book levels,1=single book), number of books per nonleaf book (book width per level),number of pages per nonleaf book, number of portlets per nonleaf page,number of pages per leaf book, and number of portlets per leaf page. Inaddition, the following options can also be specified: an upper bound onthe total number of portlets to create, whether to use short definitionand instance labels, whether to use single or multilevel menus in books,whether tree optimizations are specified and enabled, whether to share asingle pre-existing .portlet file for content or to generate one portletfile (and associated content) per portlet, number of logically identicalportal files to generate, and portal prefix.

In some embodiments, the portal generator allows automatic performancetesting on various portal configurations, which include but are notlimited to, wide but shallow portal, deep portal, and any combination ofbreath and depth. More specifically, the portal generator can beincorporated with the build phase of an application and variations canbe used to automatically run a number of tests with varying portalconfigurations. A properties file specifies some portal treecharacteristics, which include book depth, width (number pages/book),and number of portlets/page. For a non-limiting example, a wide butshallow portal, with a single shared pageflow portlet definition of 2books deep, each level is 6 books+8 pages wide (+8 portlets/page), andeach leaf level is 12 pages wide (+8 portlets/page) produces a portalwith 7 books, 80 pages, and 640 portlets. Alternatively, a huge deepportal, with lots of unshared JSP portlets and a definition of 8 booksdeep, each level is 3 books+1 page wide (and 4 portlets/page), and eachleaf level is 1 page wide (and 5 portlets/page) produces a portal with3280 books, 8747 pages, and 41,549 JSP portlets (none of them shared).In fact, a bunch of logically identical portals can be created, whichshare no resources.

High-level Run Summaries

In some embodiments, the performance testing framework provideshigh-level run summarizing capabilities to summarize, query, and analyzeresults information both within a variation and between variations inorder to bring all elements of a test run together for simplified dataanalysis. Such high-level run summaries supports run-by-run comparisonsof test and variation results, sorting and filtering of test runresults, and making it easier to identify what was going on during inthe test runs. The filtering of test run results can expose the primarydata only in order to reduce data overload and a user can alwaysdrill-down from summarized information to full set of data and alloriginal log files, results and full test configuration if so desired.

In some embodiments, , the high-level run summarizing capabilitiessupports scanning test run results for high-level patterns, providesbasis for pivot-table multidimensional results analysis, which is usedto demonstrate how changing a test setting affected performance anderrors.

In some embodiments, the content of the high-level summaries includesbut is not limited to:

-   -   summarized run information.    -   metadata of configuration properties.    -   any issues and their severity during configuration and testing,        which include but are not limited to, failed transactions,        stopped transactions, issues and their severity.    -   analyzed performance test results data, which can be saved in        the results database. For a non-limiting example, the framework        can creates and starts a load run automation controller task to        process the performance test database and create the        summary/analysis information in the results database. Results        directories can then be exposed via internet for easy external        access as discussed before.        Dynamic Analysis Window for Accurate Results Analysis

In some embodiments, the performance testing framework utilizes adynamic analysis window during the test run for accurate test resultsanalysis, which scopes results data to the most accurate data window.Such dynamic analysis window reduces error in results reported by theload generating tool. More specifically, the raw data from the loadgenerating tool results data is analyzed based on constraints to findthe window boundaries and results are reported only for the analysiswindow. For a non-limiting example, an ‘analysis window’ timeframeduring a test run is computed, which covers the period when all usershave completed at least one iteration and no users have started exiting.Testing results, which can be but are not limited to, average responsetime and standard deviation, average execute queue length, and averagethroughput in pages/sec can be generated for the analysis window.

In some embodiments, the dynamic analysis window component can be usedto identify the most accurate time window, on which results will bebased. When performance testing, often there are ramp-up, warm-up, andshutdown periods during which the system load is not constant. Thiscomponent finds the time window in which the system load is constant,specifically excluding any ramp-up, warm-up, and shutdown periods forany users. This analysis window then used during statistical resultsanalysis, and results in increased statistical results quality, lessvariance, and increased repeatability. The dynamic analysis windowduration is used as an indicator of test quality; a short test may havean empty or very short dynamic analysis window. By lengthening the testduration, the analysis window will also increase, as will quality of thetest results.

Variations Testing/Property Groups

In some embodiments, the performance testing framework allowsflexibility for specifying and varying configuration properties acrosstest runs. Some individual properties (for a non-limiting example,number of users) and associated property values can be considered alogical dimension. Other properties are typically changed in groups, anda group of properties (for a non-limiting example, various DBconfiguration settings) and their associated property values can also beconsidered a logical dimension. Combining property values for all thelogical dimensions locates a n-dimensional logical intersection point atwhich the performance test results are identified. Each logicaldimension can be represented as a ‘property group’, which specifies aset of properties which should be configured for a given token (valueassociated with a dimension), and the values associated with each of thedimensions, if not overridden, can use default preconfigured values.Here, variable testing configuration properties include but are notlimited to, number of concurrent users, execute thread count, thinktime, and memory configuration properties. For non-liming examples,values associated with the logical dimension ‘portalSize’ can be‘VerySmallPortal’, ‘SmallPortal’, ‘MediumPortal’, LargePortal’, and‘VeryLargePortal’, and values associated with another logical dimensioncan be ‘thinkTime’, with values ‘2SecondThinkTime’, etc. Property groupsand the values associated with them provide a way to map a logicalsetting, such as ‘2SecondThinkTime’, to the actual properties which mustbe configured for that logical setting to take effect when thelower-level framework targets execute.

In some embodiments, a variation of configuration is composed of a setof logical dimensions, and values associated with those dimensions to betested. The goal of a performance test is often to see how changing asingle dimension affects performance. Thus the flexibility inconfiguration can include at least the following:

-   -   dimensions to vary.    -   various values of a dimension.    -   various configuration properties that can take effect at a        dimension value.

In some embodiments, the variation testing supports specifying multipledimensions (property groups) per run. A variation specifies the logicalintersection point across multiple logical dimensions at which thesystem should be tested. For a non-limiting example, one can usestandard configuration properties for all runs except for installer anddatabase (DB) configuration properties dimensions with one run withinstaller A and Oracle DB, and another run with installer B and SybaseDB.

In some embodiments, the variation testing can change the dimensions tovary. This approach is similar to starting from a base point inn-dimensional space, and moving along different dimensions from thatbase point to see how performance varies. For non-limiting examples,with other standard configuration properties remain the same for allruns, the value of JVM, installer, and DB setting (dimension) can bevaried with at different runs. Alternatively, a user might be interestedin seeing how both user ‘thinktime’ and ‘# of concurrent users’ affectperformance, but won't need to analyze thinktime with various numbers ofusers. In this case, the user can determine this by creating a single‘base variation’, then varying thinktime, keeping the number of usersconstant, and then varying the number of users, keeping thinktimeconstant.

In some embodiments, a user may want to measure the performance acrossall possible combinations of dimensions. If a user wants to measure ifdimensions are interrelated, test combinations need to be run to measurethe interrelation, which also makes sense for capacity planning tests.For a non-limiting example, the user might be interested in seeing howboth user thinktime and ‘# of concurrent users’ affect and modeling howchanging one dimension affects another by creating ‘thinktime’ and‘numusers’ dimensions and exploring them a n-dimensional region inspace, across all dimensions. This type of testing typically results inmany runs (combinatorial explosion across dimension values), and isespecially sensitive to adding dimensions. There are at least twopossible ways to reduce the number of variations in a combinatorial set:

-   -   Try to reduce the number of dimensions involved in the        combinations. Maybe a few ‘explore deltas off a base point’        variations for a dimension can suffice instead of an ‘all        possible combinations with this dimension’.    -   If possible, reduce the number of dimensional values (property        group tokens) along a dimension. For a non-limiting example,        instead of testing with very small, small, medium, large, very        large, testing with very small, medium, and very large will        result in much fewer combinations.

In some embodiments, the variation testing supports a ‘pick and choose’model for selecting which configurations to test. Rather than use acombinatorial model (test all possible combinations of all thedimensions), such a model selects each test point. In addition,variations can be dynamically added/edited/removed while the performancetesting is running.

In some embodiments, the variation testing allows one to define his/herown dimensions for easier configuration by configuring multipleproperties of one dimension. For a non-limiting example, a number ofproperties can be set for DB type dimension (Oracle, Sybase, DB2), whichinclude but are not limited to, DB host, DB port, DB type, DB drivername and DB driver URL. In fact, pretty much any possible setting in anyphase of the performance testing can be varied, wherein suchconfiguration properties include but are not limited to, the installerto use, the application and its configuration being tested (includingportal generator properties), net (system network/process) configurationfile, database properties, server configuration properties (JVM, memory,etc), and test properties (test script, number of users, etc)

In some embodiments, the variation testing supports running multiplevariations, which are test runs with possibly different variations ofconfigurations. Multirun variation provides the foundation for fullyautomated 24×7 unattended testing, which enables a single person to runmultiple tests on multiple clusters simultaneously and shifts thebottleneck in performance testing from human effort to hardware andtime. For a non-limiting example, one can configure 50 tests, startperformance testing, and come back a few days later to examine results.

RangeFinder: Quick Rampup to Find Range

In some embodiments, a range finder can be used to find a quick rangedsnapshot of the performance of the tested system. It is an optimizationto run capacity tests of the system more quickly, especially withflatrun tests where the number of users is constant. It can find a “bestguess” range estimate of how many users the tested system can handle ina very quick timeframe. For a non-limiting example, the range finder canmeasure the performance of the tested system under 1000 concurrent usersto be within the timeframe of 0.5-1.5 second of response time.Alternatively, it can find that the tested system is operable to achieve0.5 second of response time within a load of 500-2000 concurrent users.

In some embodiments, the range finder can be used in conjunction withmultirun variations to quickly get a rough approximation of systemperformance. More specifically, the range finder utilizes a ramp-upschedule to quickly find result estimates as the # of concurrent usersis increased. It does this with a given set of inputs for the number ofusers, speed of ramp-up, and desirable response time required. Theframework then runs one test with those configuration settings and thencomputes a range of flatruns to run later. For a non-limiting example,suppose the goal is to see how many users the system can support at a 2second response time. The parameters can be as follows: up to 5000concurrent users, adding 3 users every 1 second. The framework thenwould run the tests with those parameters and return that 3000 users(for a non-limiting example) can be supported at 2 seconds or less. Athreshold should then be used to determine how wide of a range should beused based on the number 3000 users to then go and, based on what theresult is, do flatruns +/−25% of that result and space them every 50users. In this case it would be 3000 as the median 2250 as the minimumand 3750 as the maximum. The flatruns would be defined as 2250, 2300,2350 . . . up to 3750. This results in a considerable time savings ascompared to doing from 0 to 5000 every 50 (like the spread dispatcherperforms).

In some embodiments, the information from the range finder can beincorporated used by a goal navigator discussed above to zero-in moreaccurately on its goals when running a series of performance tests.Consequently, the goal navigator can find its goals more quickly sinceit starts out with more accurate capacity estimates. In addition, aseach goal navigator test run is typically quite long, the utilization ofthe range finder to provide a best guess range can be a major timesaverand save considerable amount of test time in some circumstances,especially with flatruns.

Multirun Variations/Pluggable Goal Navigator

In some embodiments, the performance testing framework can vary certaintest configuration properties automatically in a series of test runs toaccomplish a configured goal. The performance goals can include but arenot limited to, identifying the maximum number of concurrent users witha specified response time, drawing the concurrency vs. response timegraph by filling in details and reducing granularity over time till aspecified threshold is met, varying the number of users over aconfigured range by a specified increment.

In some embodiments, the performance testing framework can set up afeedback loop via a goal navigator, which analyzes data and decides whatto do next. More specifically, the goal navigator examines runs, goals,errors, and results of performance tests, and decides next setting (andconfiguration) to test with. It can accommodate multiple goals andconfigurable goal accuracy thresholds to dynamically determine whengoals have either been met or are unreachable. It can also vary multipleconfiguration properties and tolerate various degrees of errors duringthe test configurations. In addition, special error feedback allows thegoal navigator to incorporate error type, severity, and frequency in itsdetermination as whether a goal is unreachable.

In some embodiments, the variation controller has a special type ofvariation which uses the pluggable goal navigator to dynamicallyconfigure and execute zero or more runs, and dynamically determine whenthe variation (set of runs) is complete. The pluggable dispatcher can beused to dynamically identify various runs to perform, and provides afeedback loop so the dispatcher can select runs as appropriate. Thedispatcher notifies the controller when it determines no more runs arenecessary.

In some embodiments, the pluggable goal navigator supports manypluggable performance-based goals, which include but are not limited to,maximizing throughput, minimizing response time, minimizing startuptimes, and minimizing variance (maximizing consistency). There are alsosome pluggable test-based navigators which could be used to maximizeerrors, attempt to identify factors affecting reproducibility in anintermittently-failing test, and try to maximize test coverage withconfigurations (databases, platforms, etc) which have not been tested inawhile.

Goal-Based Dispatcher

In some embodiments, a specialized type of goal navigator, goal-baseddispatcher, can be utilized to focus on varying setting across runs tofind the best setting values to achieve one or more goals. Thegoal-based dispatcher is a concrete implementation of one of manypossible pluggable goal navigators, which focuses on modifying aconfiguration property to best meet response time goals. It is afocused, complex component, which handles the complicated business ofdrawing conclusions and making estimates from partially inaccurate data.Here, there is no restriction as which setting can be changed. Amultirun goal-based dispatcher can accepts multiple goals and try tofind the max number of users the system can run with while meeting eachgoal. Here, the multiple goals which can be but are not limited todifferent response times. For non-limiting examples, the goals can bethroughput or startup time, and 0.5, 1, 2, 5 second response times.

In some embodiments, the goal-based dispatcher incorporates awarenessand situational analysis in closed-loop testing when deciding what toperform in the next test run. It can automatically decide when it isdone testing, which may happen when some goals are unreachable and ithas achieved all the rest of the goals. For a non-limiting example, thegoal-based dispatcher may decide that some goals have already been metand continue working on others. External storage allows the performancetesting framework to automatically resume at any later time and restartthe goal-based dispatcher with all previous knowledge.

In some embodiments, the goal-based dispatcher may attempt to findconcurrent user sizes which match up with one or more response timegoals. Such response time goal-based dispatcher uses configured ordynamically-identified min/max values, then repeatedly selects values(number of concurrent users) to match one or more goal response timeswith concurrent user sizes. Over time, it gets closer to the goalvalues, eventually zeroing in on them, or deciding they are unreachable.For a non-limiting example, one can configure the goal-based dispatcherto find out how many concurrent users the framework can support with agiven configuration and 2, 5, and 10-second response time goals. After anumber of runs, the dispatcher may determine that 3381 users matches upwith a 2-second response time, and 4387 users matches up with a 5-secondresponse time. It may decide the 10-second response time goal isunreachable because any runs with more than 5000 users had seriouserrors (and the 10-second response time goal would require more than5000 users).

In some embodiments, the time it takes to do a run can be constrained bythe maximum number of iterations and/or the maximum variation time (themaximum time spent dynamically varying configuration to meet goals). Fora non-limiting example, if one have this set at 4 hours, but a run takes8 hours, then the performance testing framework would advance to thenext variation after the run completes. Since runs with higher responsetimes generally take much longer to run (linear increase in run time),it's generally better to under-shoot the top goal response time thanovershoot it. Additionally, it helps if the exploration is not extremelyaggressive—this avoids significantly overshooting the correct number ofusers.

In some embodiments, the goal-based dispatcher periodically reports itsgoal assessment by writing results to a database table. The results (howmany users the dispatcher thinks each response time goal supports) canbe viewed with an spreadsheet while the tests are running. In addition,the goal-based dispatcher refines its best-fit estimates ofconfiguration properties for each goal. By sharing this best-fit data asit progresses, the goal-based dispatcher supports a usage model in whichthe system can (given enough time) find the optimal settings, but can bestopped at any point if one decides the system resources should be usedfor another purpose. If at a later point, additional accuracy isdesired, the testing can be restarted and the goal-based dispatcher willpick up where it left off and continue refining its best-fit estimatesfor each goal.

In some embodiments, the goal-based dispatcher includes goal accuracyconfiguration and error severity/count configuration. It assesseswhether a run has errors. while performing test runs. Its error run (runwith error) assessment is configurable at least in the following ways:

-   -   Error runs can either be ‘hard’ (no response time data,        generally due to too many transaction failures), or ‘soft’        (response time data, but there were other issues).    -   Error runs can add an upper error bound to the explored area. In        other words, an error run (if repeated enough times at a value)        will cause the dispatcher to set an upper limit on the number of        users. This may cause the dispatcher to decide certain goals are        unreachable due to this limit. One can specify the number of        times an error run (hard or soft) must be found at a value        (number of users) before the value becomes an upper bound.    -   One can configure what types of issues will cause a run to be        considered a soft error run. For a non-limiting example, one can        require there to be at least 50 failed transactions in a run        before it is considered a soft error run.

In some embodiments, the goal-based dispatcher can be configured toanalyze the run data, even if it has errors. The framework can beconfigured to log all errors (so the dispatcher can see them) but notthrow exceptions it detects, except for the really serious ones. Theerror repeatability threshold, run handling and soft failureconfiguration properties can also be configured.

In some embodiments, the goal-based dispatcher works in at least thefollowing objectives, each of which is configurable:

-   -   Bound response time goal values (find one response time below        the minimum goal, and another above the maximum goal). The        dispatcher can start with the configured min/max hints and        explore if needed.    -   Select values at evenly-spaced intervals along the global range,        in order to get a good global best-fit line. The number of        values (unique number of users) to try in the global range is        configurable.    -   Ensure the global range best-fit line quality meets a quality        threshold if configured.    -   Divide the global-range into sub-ranges by drilling down based        on what appears to be the most optimal sub-range for each goal.        Each sub-range has a configurable minimum number of values, and        a configurable best-fit line quality threshold.    -   Refine the most optimal goal sub-range till it meets the quality        thresholds.    -   Find a final goal sub-range, with a configurable max size, and        configurable number of tested values.    -   Monitor the best-fit line quality over a configurable number of        runs, ensuring it changes by less than a configurable        percentage.        The goal-based dispatcher considers a response time goal as        complete when it gets through the final objective. It can also        dynamically re-evaluate each objective each time it receives new        data, and will re-attempt to meet a previously-met objective if        data indicates the objective is no longer being met. This        supports usage in a testing environment with a high degree of        variance.        Optimizing Explorer

In some embodiments, an optimizing explorer can be used as a specializedgoal navigator to determine the optimal configurations and configurationboundaries of a tested system. As the tested system runs, it continuallyupdates/reports the ‘best known configuration properties so far duringthe performance test. It adopts the “start it and forget it” model,which will save the best configuration properties for a tested systembased on the performance testing results and report the optimalconfiguration when the user comes back in a few weeks.

In some embodiments, optimizing explorer can be configured with thefollowing:

-   -   A starting point (group of configuration properties) of the        system configuration.    -   Performance metric (how to measure success) of the tested        system, which includes one or more of: maximum throughput,        average response time, minimum startup time, and average number        of concurrent users.    -   a set of one or more configuration properties to keep constant        (implicitly).    -   a set of one or more configuration properties to vary and how        much to vary each of the set of one or more configuration        properties.    -   when to stop varying the configuration properties, e.g., less        than 0.05% performance increase in any dimension or after 5000        test runs.

In some embodiments, the optimizing explorer is operable to start aseries of test runs from the starting point and moving across thevarious dimensions of the configuration space. It identifies the optimaldimension(s) to change, and then moves in the direction to increaseperformance and re-tests at the current point in space if needed. Morespecifically, it records the test data externally so it can restart atany time and pick up where it left off. As the optimizing explorerlearns about the performance of the system within its configurationspace, it records correlation information indicating how much change inany given dimension affects the performance at a given point. From thisinformation, the optimizing explorer builds a correlation model toidentify which factors had the largest affect on performance across theconfiguration space. In addition, the optimizing explorer alsoidentifies validity boundaries of the configuration space as it recordscertain points in the space where some configuration properties becomeinvalid.

In some embodiments, the optimizing explorer is operable to periodicallyre-evaluate its performance data at each point in configuration space,determines if additional performance probing on one or more dimensionsat that point is necessary, then selects the next point to move to.

Dimensional Explorer

In some embodiments, each configuration setting can be considered as a‘dimension’, and a variation composed of N configuration properties canthen be represented as a point in N-dimensional configuration space.From this point in space, one configuration setting is adjusted by anymove in a single dimension. For a non-limiting example, if one movesalong the ‘user think time’ dimension while keeping all otherconfiguration properties (installer version, portal size, number ofconcurrent users, etc) constant, the user think time setting (and thissetting only) is changed.

In some embodiments, a dimensional explorer can be used as a specializedgoal navigator to analyze one or more of: how various configuration(logical) properties affect performance at a given point, how sensitivethe configuration properties are to change, and report this information.More specifically, the dimensional explorer is operable to perform aseries of runs to find and report cost/benefit information bynavigating/moving across multiple dimensions from a starting point inthe space (group) of configuration properties. It measures success(performance metric) of the performance test based on one or more ofperformance measures of the tested system: maximum throughput, averageresponse time, minimum startup time, and average number of concurrentusers. In some embodiments, the dimensional explorer provides at leastthe following primary features:

-   -   From a given configuration (c), how varying each of (n)        explicitly specified properties over a specified range will        affect performance. A single explicitly-specified property is        varied at a time.    -   For each explicitly specified property (p) which is varied,        which range of values of (p) was valid. Some settings of (p) may        cause failures in the tested system.    -   For each explicitly specified property (p) which is varied, over        the valid range of values of (p), how sensitive the system        performance was to changes in property (p). Some properties may        have a large effect on performance; others may have no effect.        The sensitivity is weighted, so values closer to (c) are        weighted more heavily.

In some embodiments, the dimensional explorer can be configured with thefollowing:

-   -   a set of one or more configuration properties to keep constant        (implicitly).    -   a set of one or more configuration properties to vary.    -   how much to vary each of the set of one or more configuration        properties.    -   when to stop varying the configuration properties, e.g., after        200 test runs.

In some embodiments, the dimensional explorer records its dataexternally so it can be restarted at any time and pick up where it leftoff. As the explorer learns about the performance of the tested systemwith its space of configuration properties, it records correlationinformation indication how much changing any given dimension affects theperformance of the tested system. In the meantime, it also learns howsensitive the specified performance metric is to a given config setting,and uses this information when changing each setting. Some configurationproperties, such as number of servers, are very sensitive to changeswherein tiny changes can result in large performance changes. Fornon-limiting examples, varying the number of servers from 2-4 can have aperformance impact of 180% [0% to 180%] and varying the number ofexecute threads from5-50 can have a performance impact of 30% [-−20% to+10%]. In contrast, some other configuration properties, such as cachesize (measured in bytes), are very insensitive to changes as hugechanges need to be made before performance differences are detectable.For non-limiting examples, varying the cache size from 1000-500,000bytes can have a performance impact of 5% [−2% to +3%] and varyingsession replication can b have a performance impact of 0% [0% to 0%].Such information on configuration sensitivity can either be specified ordetermined automatically as the test runs.

In some embodiments, the dimensional explorer measures and toleratesdata variability across the space of the con figuration propertiesduring performance tests. For a non-limiting example, test results maynormally vary by 5% and the dimensional explorer can periodicallymeasure such data variation, incorporate it in its analysis, and reportsit in its results.

Queued/On-Demand Testing

In some embodiments, the performance testing framework supports runningtests across multiple test sets as resources become available,separating the configuration the test sets from the execution of thetest sets. Here, each test includes all phases of testing, which includebut are not limited to, fetch and run a platform installer; configureand build the application; configure domain, server script directories,and proxy server; and configure and run test scenario. Such an approachincreases resource utilization and test productivity, making maximumtest parallelism possible.

In some embodiments, the performance testing framework incorporatesrunning multiple test sets simultaneously across resources. For anon-limiting example, four test sets can be run across 50 machines inparallel. In addition, resource management and resource classes can beincorporated, wherein each test run reserves resources for the durationof the test run and releases them when the test run completes. If a testis unable to execute because not enough resources are available, theframework will try to run other tests instead. Resources can also beassigned to a <test set>, which means other test sets cannot reservethem. For a non-limiting example, one large capacity test may require 40machines, which leaves only 10 for all other tests that also want torun.

One embodiment may be implemented using a conventional general purposeor a specialized digital computer or microprocessor(s) programmedaccording to the teachings of the present disclosure, as will beapparent to those skilled in the computer art. Appropriate softwarecoding can readily be prepared by skilled programmers based on theteachings of the present disclosure, as will be apparent to thoseskilled in the software art. The invention may also be implemented bythe preparation of integrated circuits or by interconnecting anappropriate network of conventional component circuits, as will bereadily apparent to those skilled in the art.

One embodiment includes a computer program product which is a machinereadable medium (media) having instructions stored thereon/in which canbe used to program one or more computing devices to perform any of thefeatures presented herein. The machine readable medium can include, butis not limited to, one or more types of disks including floppy disks,optical discs, DVD, CD-ROMs, micro drive, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices,magnetic or optical cards, nanosystems (including molecular memory ICs),or any type of media or device suitable for storing instructions and/ordata. Stored on any one of the computer readable medium (media), thepresent invention includes software for controlling both the hardware ofthe general purpose/specialized computer or microprocessor, and forenabling the computer or microprocessor to interact with a human user orother mechanism utilizing the results of the present invention. Suchsoftware may include, but is not limited to, device drivers, operatingsystems, execution environments/containers, and applications.

The foregoing description of the preferred embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Many modifications andvariations will be apparent to the practitioner skilled in the art.Particularly, while the concept “controller” is used in the embodimentsof the systems and methods described above, it will be evident that suchconcept can be interchangeably used with equivalent concepts such as,class, method, type, interface, bean, component, object model, and othersuitable concepts. While the concept “components” is used in theembodiments of the systems and methods described above, it will beevident that such concept can be interchangeably used with equivalentconcepts such as, bean, class, method, type, process, object model, andother suitable concepts. While the concept “configuration” is used inthe embodiments of the systems and methods described above, it will beevident that such concept can be interchangeably used with equivalentconcepts such as, property, attribute, annotation, field, element, andother suitable concepts. Embodiments were chosen and described in orderto best describe the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention, the various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalents.

1. A system to support performance testing, comprising: a tested system, comprising one or more of: one or more application servers operable to deploy and provide an application to a user; and a load generating tool operable to execute a plurality of test runs to measure performance of the application automatically; an infrastructure, comprising one or more of: an automation controller operable to analyze performance data of the plurality of test runs within and/or between a plurality of variations; and a variation controller operable to: specify the plurality of variations of configuration properties of the tested system, wherein each of the plurality of variations is composed of a set of logical dimensions and values associated with each of those dimensions to be tested; and generate and execute the plurality of test runs for the plurality of variations.
 2. The system according to claim 1, wherein: the application is a Web-based application.
 3. The system according to claim 1, wherein: configuration properties of the tested system include one or more of: database type, server configuration, network configuration, application configuration, and test configuration.
 4. The system according to claim 1, wherein: the values associated with each of those dimensions is a set of preconfigured properties and associated values.
 5. The system according to claim 1, wherein: a variation controller is operable to add, edit, remove, or modify one of the plurality of variations dynamically while the test run for the variation is running and using the variation.
 6. The system according to claim 1, wherein: the values associated with each of the dimensions, if not overridden, use default preconfigured values.
 7. The system according to claim 1, wherein: the set of dimensions and/or the values associated with one or more of the dimensions are user-defined, wherein explicitly specified values of the one or more of the dimensions override their default values.
 8. The system according to claim 1, wherein: performance of the application is measured by one or more of: response time, throughput, capacity, and scalability.
 9. The system according to claim 8, wherein: the plurality of variations starts from a base point in n-dimensional space of the set of dimensions, and moves along different dimensions in the set from that base point to evaluate how performance of the application varies.
 10. The system according to claim 8, wherein: goal of one of more of the plurality of test runs is to determine how change in one of the set of logical dimensions affects performance of the application.
 11. The system according to claim 1, wherein: the plurality of test runs is used to measure interrelation among the set of dimensions.
 12. A method to support performance testing, comprising: specifying a plurality of variations of configuration properties of a tested system, wherein each of the plurality of variations is composed of a set of logical dimensions and values associated with each of those dimensions to be tested; generating and executing a plurality of test runs for the plurality of variations automatically to measure performance of the tested system; and analyzing performance data of the plurality of test runs within and/or between the plurality of variations.
 13. The method according to claim 12, further comprising: adding, editing, removing or modifying one of the plurality of variations dynamically while the test run for the variation is running and using the variation.
 14. The method according to claim 12, further comprising: allowing the set of dimensions and/or the values associated with one or more of those dimensions to be defined by a user, wherein explicitly specified values of the one or more of the dimensions override their default values.
 15. The method according to claim 12, further comprising: measuring the performance of the tested system by one or more of: response time, throughput, capacity, and scalability.
 16. The method according to claim 15, further comprising one or more of: starting the plurality of variations from a base point in n-dimensional space of the set of dimensions, and moving it along different dimensions in the set from that base point to evaluate how performance of the application varies; and determining how change in one of the set of logical dimensions affects performance of the application via one or more of the plurality of test runs.
 17. The method according to claim 12, further comprising: measuring interrelation among the set of dimensions via the plurality of test runs.
 18. The method according to claim 12, further comprising one or more of: summarizing and reporting analysis result of the performance data to a user; presenting the analysis result to the user on a Web browser; and exporting the analysis result to a spreadsheet.
 19. A machine readable medium having instructions stored thereon that when executed cause a system to: specify a plurality of variations of configuration properties of a tested system, wherein each of the plurality of variations is composed of a set of logical dimensions and values associated with each of those dimensions to be tested; generate and execute a plurality of test runs for the plurality of variations automatically to measure performance of the tested system; and analyze performance data of the plurality of test runs within and/or between the plurality of variations.
 20. A system to support performance testing, comprising: means for specifying a plurality of variations of configuration properties of a tested system, wherein each of the plurality of variations is composed of a set of logical dimensions and values associated with each of those dimensions to be tested; means for generating and executing a plurality of test runs for the plurality of variations automatically to measure performance of the tested system; and means for analyzing performance data of the plurality of test runs within and/or between the plurality of variations. 